Electronic apparatus with eye calibration and method for controlling the same

ABSTRACT

An electronic apparatus includes a control unit that performs control to acquire first orientation information indicating a first orientation of the body and first line-of-sight information about the user&#39;s line-of-sight, of a case where a first eye tracking calibration display is displayed on the screen, acquire second orientation information indicating a second orientation of the body different from the first orientation and second line-of-sight information about the user&#39;s line-of-sight, of a case where a second eye tracking calibration display is displayed on the screen, and cause, in a case where the second orientation information is different from the first orientation information, the screen to display a predetermined display that prompts the user to change the orientation of the body to the first orientation.

BACKGROUND Field

The present disclosure relates to an electronic apparatus capable of aline-of-sight input by a user's line-of-sight and a method forcontrolling the electronic apparatus.

Description of the Related Art

A conventional camera is known to detect a line-of-sight of aphotographer (user), detect a position (region) in a finder of thecamera currently being gazed by the photographer, and control an imagingfunction, such as automatic focus adjustment. However, accuracy of theuser's line-of-sight detection depends on a pupil diameter of a user'seye, how the user looks into the finder, and an ambient brightness.Therefore, it is necessary to acquire the user's line-of-sight as dataand then perform eye tracking calibration to obtain a calibration valuefor calculating a user's line-of-sight position, i.e., the positioncurrently being gazed by the user. Performing calibration reduces adifference between a position currently being gazed by the user and acalculated line-of-sight position corresponding to the user'sline-of-sight. This enables the user to specify a position and execute afunction by using the line-of-sight without feeling a sense ofstrangeness. Japanese Patent Application Laid-Open No. 07-255676discusses a technique for detecting an orientation (horizontal orvertical position) of an optical apparatus at the start of calibration,and storing a calibration value calculated based on the detectedorientation and user's line-of-sight information (line-of-sight data),as calibration data.

However, in a case where line-of-sight data as a plurality of pieces ofline-of-sight information for generating calibration data is acquired,the technique disclosed in Japanese Patent Application Laid-Open No.07-255676 acquires only an orientation of the optical apparatus at thestart of calibration. For each of a plurality of detection points, thetechnique does not consider each orientation of the optical apparatusfor the second and subsequent detection points at timings of acquiringthe line-of-sight data. If an orientation of the optical apparatus at atiming of acquiring line-of-sight data for the first point (index 1) isdifferent from an orientation of the optical apparatus at a timing ofacquiring line-of-sight data for the second point (index 2), accuracy ofthe line-of-sight data will be degraded. This accuracy degradationaffects calibration data including a calibration value to be calculated.This causes a difference between the position currently being gazed bythe user and the line-of-sight position corresponding to the user'sline-of-sight.

SUMMARY

Various embodiments of the present disclosure provide for generatingcalibration data having higher accuracy.

According to various embodiments of the present disclosure, anelectronic apparatus includes a detection unit configured to detect anorientation of a body including an acquisition unit, and at least onememory and at least one processor which function as an acquisition unitconfigured to acquire information about a user's line-of-sight to ascreen, and a control unit configured to perform control to execute aneye tracking calibration, based on the acquired information about theuser's line-of-sight, wherein the control unit performs control toacquire first orientation information indicating a first orientation ofthe body and a first line-of-sight information about the user'sline-of-sight, of a case where a first eye tracking calibration displayis displayed on the screen, acquire second orientation informationindicating a second orientation of the body and a second line-of-sightinformation about the user's line-of-sight, of a case where a second eyetracking calibration display is displayed on the screen, and cause, in acase where the second orientation information is different from thefirst orientation information, the screen to display a predetermineddisplay that prompts the user to change the orientation of the body tothe first orientation.

Further features of the present disclosure will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B are diagrams each illustrating an outer appearance of adigital camera according to one embodiment.

FIG. 2 is a block diagram illustrating a configuration of the digitalcamera according to one embodiment.

FIG. 3 is a control flowchart illustrating calibration processingaccording to one embodiment.

FIGS. 4A and 4B are control flowcharts illustrating data acquisitionprocessing in the calibration processing according to one embodiment.

FIGS. 5A to 5H are diagrams illustrating example screens that aredisplayed at acquiring of line-of-sight data as line-of-sightinformation according to one embodiment.

FIGS. 6A to 6C are diagrams illustrating orientations of the digitalcamera in the data acquisition processing and positional relationsbetween a line-of-sight detection block and an eye according to oneembodiment.

FIG. 7 is a diagram illustrating an example display of a setting screenrelated to the line-of-sight according to one embodiment.

DESCRIPTION OF THE EMBODIMENTS

An exemplary embodiment of the present disclosure will be described indetail below with reference to the accompanying drawings.

It is to be noted that the following exemplary embodiment is merely oneexample for implementing the present invention and can be appropriatelymodified or changed depending on individual constructions and variousconditions of apparatuses to which embodiments of the present inventionare applied. Thus, the present invention is in no way limited to thefollowing exemplary embodiment.

FIGS. 1A and 1B are diagrams each illustrating an outer appearance of adigital camera 100 as an example of an apparatus (electronic apparatus)to which various embodiments of the present disclosure can be applied.FIG. 1A is a perspective view illustrating a front face of the digitalcamera 100, and FIG. 1B is a perspective view illustrating a rear faceof the digital camera 100. Referring to FIGS. 1A and 1B, a display unit28 is disposed on the rear face of the digital camera 100 to displayimages and various information. A touch panel 70 a is a touch operationmember for detecting a touch operation on a display surface (operationsurface) of the display unit 28. An outside-finder liquid crystaldisplay unit 43 disposed on a top face of the digital camera 100displays various setting values of the camera including a shutter speedand an aperture value.

A shutter button 61 is an operation member for issuing an imagecapturing instruction. A mode selection switch 60 is an operation memberfor selecting various modes. Terminal covers 40 protect connectors (notillustrated) each for connecting a connection cable from an externalapparatus to the digital camera 100. A main electronic dial 71 includedin an operation unit 70 is a rotary operation member that is used(turned) to change setting values, such as a shutter speed and anaperture value. A power switch 72 included in the operation unit 70 isan operation member for turning power of the digital camera 100 ON andOFF. A sub electronic dial 73 included in the operation unit 70 is arotary operation member for moving a selection frame and feeding images.A cross key 74 included in the operation unit 70 is an operation memberhaving buttons that can be pressed in four different directions. Thecross key 74 is to enable operations each corresponding to a differentone of pressed directions to be executed. A SET button 75 included inthe operation unit 70 is a push button mainly used to determine aselection item. A moving image button 76 included in the operation unit70 is an operation member that is used to issue instructions forstarting and stopping moving image capturing (recording).

An M-Fn button 77 included in the operation unit 70 is used (pressed) inan image capturing standby state to quickly change various settings,such as the white balance and ISO sensitivity. The M-Fn button 77 can beassigned various functions as well as the image capturing standby stateby a user. The M-Fn button 77 can be assigned a line-of-sight inputfunction. In the image capturing standby state, for example, theline-of-sight input function can be turned ON or OFF by pressing theM-Fn button 77. The M-Fn button 77 is also used during eye trackingcalibration for more accurate detection of a position currently beinggazed by the user, by using a line-of-sight detection block 160(described below). In a case where the user gazes a certain gaze pointin a calibration display currently displayed and presses the M-Fn button77, the pressing of the M-Fn button 77 functions as a determinationoperation. Calibration will be described in detail below.

A playback button 79 included in the operation unit 70 switches betweenan image capturing mode and a playback mode. In a case where the userpresses the playback button 79 in the image capturing mode, the digitalcamera 100 enters the playback mode and the latest image among imagesrecorded in a recording medium 200 can be displayed on the display unit28. A menu button 81 included in the operation unit 70 is used (pressed)to display a menu screen on the display unit 28 on which the user canperform various settings. The user can intuitively perform varioussettings by using the menu screen displayed on the display unit 28, thecross key 74, and the SET button 75.

A communication terminal 10 is used by the digital camera 100 tocommunicate with a lens unit 150 attachable to and detachable from thedigital camera 100 (described below). An eyepiece 16 is an eyepiecemember of an eyepiece finder (look-in finder) of the digital camera 100.The user can visually recognize an image displayed in an Electric ViewFinder (EVF) 29 provided inside the finder through the eyepiece 16. In acase where an optical image of a subject (subject image) can be capturedthrough the lens unit 150, the user can visually recognize an opticalimage (subject image) of the subject by using the internal opticalfinder through the eyepiece 16. An eye-contact detection unit 57 is aneye-contact detection sensor for detecting whether the user's eye is incontact with the eyepiece 16. A cover 202 covers a slot that stores therecording medium 200. A grip portion 90 has a shape that is easilygripped with a user's right hand when the user holds the digital camera100. The shutter button 61 and the main electronic dial 71 are disposedat positions where an operation can be performed by a forefinger of theuser's right hand in a state where the user holds the digital camera 100by gripping the grip portion 90 with a little finger, a third finger,and a middle finger of the user's right hand. The sub electronic dial 73is disposed at a position where an operation can be performed by a thumbof the user's right hand in the same state.

FIG. 2 is a block diagram illustrating an example configuration of thedigital camera 100 and the lens unit 150 as an electronic apparatusaccording to the present exemplary embodiment. Referring to FIG. 2 , thelens unit 150 mounts an interchangeable imaging lens to capture asubject image. While a lens 103 normally includes a plurality of lenses,FIG. 2 illustrates a single lens as the lens 103 for simplification. Acommunication terminal 6 is used by the lens unit 150 to communicatewith the digital camera 100. The lens unit 150 communicates with asystem control unit 50 via the communication terminal 6 and theabove-described communication terminal 10. A lens system control circuit4 controls a diaphragm 1 via a diaphragm drive circuit 2. The lenssystem control circuit 4 focuses on the subject by displacing the lens103 via an Automatic Focus (AF) drive circuit 3.

A shutter 101 is a focal plane shutter capable of freely controlling anexposure time of an imaging unit 22 under control of the system controlunit 50.

The imaging unit 22 is an image sensor including a Charge Coupled Device(CCD) or Complementary Metal Oxide Semiconductor (CMOS) sensor forconversing an optical image into an electrical signal. Ananalog-to-digital (A/D) converter 23 converts the analog signal outputfrom the imaging unit 22 into a digital signal.

An image processing unit 24 subjects data from the A/D converter 23 ordata from a memory control unit 15 to predetermined pixel interpolation,resizing processing, such as reduction, and color conversion processing.The image processing unit 24 also performs predetermined calculationprocessing by using captured image data. The system control unit 50performs exposure control and distance measurement control, based on thecalculation result obtained by the image processing unit 24. Thisenables the digital camera 100 to perform AF processing, AutomaticExposure (AE) processing, and Electronic Flash Preliminary Emission (EF)processing based on the Through-The-Lens (TTL) method. The imageprocessing unit 24 also performs predetermined calculation processing byusing the captured image data and performs TTL-based Automatic WhiteBalance (AWB) processing, based on the obtained calculation result.

The memory control unit 15 controls data communication between the A/Dconverter 23, the image processing unit 24, and a memory 32. Data outputfrom the A/D converter 23 is written in the memory 32 via the imageprocessing unit 24 and the memory control unit 15 or directly written inthe memory 32 via the memory control unit 15. The memory 32 stores imagedata captured by the imaging unit 22 and converted into digital data bythe A/D converter 23, and image data to be displayed on the display unit28 and the EVF 29. The memory 32 is provided with a sufficient storagecapacity to store a predetermined number of still images, and movingimages and sound for a predetermined time period.

The memory 32 also serves as an image display memory (video memory). Thedisplay image data written in the memory 32 is displayed on the displayunit 28 and the EVF 29 via the memory control unit 15. The display unit28 and the EVF 29 display data on a liquid crystal display (LCD) or anorganic electroluminescence display according to a signal from thememory control unit 15. Live view display (LV display) can be performedby successively transferring data A/D-converted by the A/D converter 23and stored in the memory 32, to the display unit 28 or the EVF 29.Hereinafter, an image displayed in the live view is referred to as alive view image (LV image).

A system timer 53 is a time measurement unit for measuring time to beused for various control and time of a built-in clock.

The mode selection switch 60, a first shutter switch 62, a secondshutter switch 64, and the operation unit 70 are operation members forinputting various operation instructions to the system control unit 50.The mode selection switch 60 switches an operation mode of the systemcontrol unit 50 between a still image capturing mode, a moving imagecapturing mode, and a playback mode. The still image capturing modeincludes an automatic image capturing mode, an automatic scenedetermination mode, a manual mode, a diaphragm priority mode (Av mode),a shutter speed priority mode (Tv mode), and a program AE mode (P mode).The still image capturing mode further includes various scene modes asimaging settings each for a different image capturing scene, a custommode, and the like. The mode selection switch 60 enables the user todirectly select any one of these modes. Alternatively, the modeselection switch 60 may display an image capturing mode list screen, andthe user may select any one of the plurality of displayed modes andchange the mode by using other operation members. Likewise, the movingimage capturing mode may also include a plurality of modes.

The first shutter switch 62 turns ON in the middle of depression of theshutter button 61 provided on the digital camera 100, what is calledhalfway pressing (image capturing preparation instruction), to generatea first shutter switch signal SW1. The first shutter switch signal SW1causes the system control unit 50 to start an imaging capturingpreparation operation, such as the AF processing, the AE processing, theAWB processing, and the EF processing.

The second shutter switch 64 turns ON upon completion of an operation ofthe shutter button 61, what is called full pressing (image capturinginstruction), to generate a second shutter switch signal SW2. Uponissuance of the second shutter switch signal SW2, the system controlunit 50 starts a series of operations for imaging processing rangingfrom reading of a signal from the imaging unit 22 to writing capturedimage as an image file in the recording medium 200.

The operation unit 70 includes various operation members as input unitsfor receiving operations from the user. The operation unit 70 includesat least the following operation members: the shutter button 61, thetouch panel 70 a, the main electronic dial 71, the power switch 72, thesub electronic dial 73, the cross key 74, the SET button 75, the movingimage button 76, the M-Fn button 77, the playback button 79, and themenu button 81.

The touch panel 70 a and the display unit 28 can be integrally formed.For example, the touch panel 70 a is configured to be a panel havinglight-transmissivity which does not disturb display on the display unit28 and is disposed on an upper layer of the display surface of thedisplay unit 28. Then, input coordinates on the touch panel 79 a areassociated with display coordinates on the display screen of the displayunit 28. This enables providing such a graphical user interface (GUI)that virtually allows the user to directly operate the screen displayedon the display unit 28. The system control unit 50 can detect thefollowing operations on the touch panel 70 a and states thereof.

-   -   An operation to start touching the touch panel 70 a with a        finger or a pen that has not been in contact with the touch        panel 70 a (hereinafter referred to as a “touch-down”)    -   A state where the finger or the pen is in contact with the touch        panel 70 a (hereinafter referred to as a “touch-on”)    -   An operation to move the finger or the pen while in contact with        the touch panel 70 a (hereinafter referred to as a “touch-move”)    -   An operation to detach the finger or the pen in contact with the        touch panel 70 a from the touch panel 70 a to end touching        (hereinafter referred to as a “touch-up”)    -   A state where the finger or the pen is out of contact with the        touch panel 70 a (hereinafter referred to as a “touch-off”)

When a touch-down is detected, a touch-on is also detected at the sametime. After the touch-down, the touch-on is normally kept being detecteduntil a touch-up is detected. A touch-move is detected in a state wherethe touch-on is detected. Even when the touch-on is detected, atouch-move is not detected if the touch position is not moving. After atouch-up is detected for all of fingers or the pen that have been incontact with the touch panel 70 a, a touch-off is detected.

The above-described operations and states as well as positioncoordinates of a position where a finger or a pen contacts the touchpanel 70 a are notified to the system control unit 50 via an internalbus. Based on the notified information, the system control unit 50determines what kind of operation (touch operation) has been performedon the touch panel 70 a. For a touch-move, the moving direction of thefinger or the pen moving on the touch panel 70 a can be determined foreach of vertical and horizontal components on the touch panel 70 a,based on changes of the position coordinates. In a case where atouch-move by a predetermined distance or longer is detected, the systemcontrol unit 50 determines that a slide operation has been performed. Anoperation to quickly move the finger by a certain distance while incontact with the touch panel 70 a and then releasing the finger from thetouch panel 70 a is referred to as a flick. In other words, a flick isan operation to quickly flip the surface of the touch panel 70 a withthe finger. In a case where a touch-move at a predetermined speed orhigher by a predetermined distance or longer is detected and then atouch-up is subsequently detected, it can be determined that a flick hasbeen performed (a flick has been performed following a slide). A touchoperation to simultaneously touch a plurality of positions (for example,two positions) and bring these positions close to each other is referredto as a “pinch-in”. A touch operation to move these positions away fromeach other is referred to as a “pinch-out”. A pinch-out and a pinch-inare collectively referred to as a pinch operation (or simply referred toas a “pinch”). The touch panel 70 a may be of any one of various typesincluding a resistance film type, a capacitance type, a surface elasticwave type, an infrared type, an electromagnetic induction type, an imagerecognition type, and an optical sensor type. A touch is detected whenthe finger or the pen comes into contact with the touch panel 70 a orwhen the finger or the pen comes close to the touch panel 70 a dependingon the type, and either type is applicable.

A power source control unit 80 including a battery detection circuit, adirect-current to direct-current (DC-DC) converter, and a switch circuitfor selecting a block to be supplied with power detects presence orabsence of a battery, the battery type, and the remaining batterycapacity. The power source control unit 80 also controls the DC-DCconverter, based on the detection result and an instruction from thesystem control unit 50 to supply required voltages to the recordingmedium 200 and other components for required time periods. A powersource unit 30 includes a primary battery (such as an alkaline batteryand lithium battery), a secondary battery (such as a NiCd battery, NiMHbattery, and Li battery), and an alternating current (AC) adaptor.

A recording medium interface (I/F) 18 is an interface to the recordingmedium 200, such as a memory card and a hard disk. The recording medium200 is, for example, a memory card for recording captured images,including a semiconductor memory and a magnetic disk.

A communication unit 54 performs wireless or wire cable connection totransmit and receive an image signal and an audio signal. Thecommunication unit 54 is connectable with a wireless Local Area Network(LAN) and the Internet. The communication unit 54 can communicate withan external apparatus by using Bluetooth® and Bluetooth® Low Energy. Thecommunication unit 54 is able to transmit images (including live viewimage) captured by the imaging unit 22 and images stored in therecording medium 200, and receive images and other various informationfrom an external apparatus.

A nonvolatile memory 56 is an electrically erasable recordable memory,such as a Flash read only memory (ROM). Constants, programs, and thelike for operations of the system control unit 50 are stored in thenonvolatile memory 56. Programs stored in the nonvolatile memory 56refer to computer programs for executing various flowcharts (describedbelow) according to the present exemplary embodiment.

The system control unit 50 is a control unit including at least oneprocessor or circuit, and controls the entire digital camera 100. Eachpiece of processing according to the present exemplary embodiment(described below) is implemented when the system control unit 50executes the above-described programs recorded in the nonvolatile memory56. A system memory 52 is, for example, a random access memory (RAM).Constants and variables for operations of the system control unit 50 andprograms read from the nonvolatile memory 56 are loaded into the systemmemory 52. The system control unit 50 also performs display control bycontrolling the memory 32, the display unit 28, and the like.

An orientation detection unit 55 detects information about anorientation of a body of the digital camera 100 in the gravitydirection. An acceleration sensor and a gyroscope sensor can be used asthe orientation detection unit 55. Motions of the digital camera 100(pan, tilt, raising, and stand still) can also be detected by using theacceleration sensor and the gyroscope sensor serving as the orientationdetection unit 55. Based on the orientation detected by the orientationdetection unit 55, the system control unit 50 can determine whether theimage captured by the imaging unit 22 is an image captured with thedigital camera 100 horizontally held or an image captured with thedigital camera 100 vertically held. The system control unit 50 canappend orientation information corresponding to an orientation detectedby the orientation detection unit 55 to an image file of an imagecaptured by the imaging unit 22 or rotate the image before recording.

In the calibration described below, it is important to grasp apositional relation between the digital camera 100 and an eye 161. Thus,in a case where user's line-of-sight data is acquired as line-of-sightinformation by the calibration, the system control unit 50 also acquiresan orientation of the digital camera 100 detected by the orientationdetection unit 55 and stores the acquired orientation as the orientationinformation. This enables more accurate determining of a line-of-sightposition corresponding to a line-of-sight even in a case where the userperforms image capturing at various angles.

The line-of-sight detection block 160 detects a line-of-sight of the eye(user's eyeball) 161 in contact with the eyepiece 16. The line-of-sightdetection block 160 includes a dichroic mirror 162, an imaging lens 163,a line-of-sight detection sensor 164, infrared emitting diodes 166, anda line-of-sight detection circuit 165.

The infrared emitting diodes 166 serve as light emitting elements fordetecting a user's line-of-sight in the inside-finder display unit andirradiate the eye (user's eyeball) 161 in contact with the eyepiece 16with infrared light. The infrared light emitted from the infraredemitting diodes 166 is reflected on the eye (user's eyeball) 161 and thereflected infrared light reaches the dichroic mirror 162. The dichroicmirror 162 reflects only infrared light and transmits visible light. Thereflected infrared light with the changed optical path forms an image onan imaging plane of the line-of-sight detection sensor 164 via theimaging lens 163. The imaging lens 163 is an optical member thatconfigures a line-of-sight detection optical system. The line-of-sightdetection sensor 164 includes an imaging device, such as a CCD imagesensor.

The line-of-sight detection sensor 164 electrically converts theincident infrared reflected light into an electrical signal and outputsthe electrical signal to the line-of-sight detection circuit 165. Theline-of-sight detection circuit 165 including at least one processordetects the user's line-of-sight from an image or movement of the eye(user's eyeball) 161, based on a signal output from the line-of-sightdetection sensor 164, and outputs detection information to the systemcontrol unit 50.

According to various embodiments of the present disclosure, aline-of-sight is detected based on a method called a cornea reflectionmethod by using the line-of-sight detection block 160. The corneareflection method detects an orientation and a position of aline-of-sight, based on a positional relation between reflected light(infrared light emitted from the infrared emitting diodes 166 andreflected by a cornea of the eye (eyeball) 161) and a pupil of the eye(eyeball) 161. Other various methods for detecting an orientation and aposition of a line-of-sight include a sclera reflection method thatutilizes a difference in light reflectance between black and white eyeregions. Other methods for detecting a line-of-sight are also applicableas long as the orientation and position of a line-of-sight can bedetected.

The user's line-of-sight detected in the line-of-sight detection block160 is used by the digital camera 100, for example, as a function ofdisplaying an AF frame at the line-of-sight position corresponding tothe user's line-of-sight and performing AF. The AF frame moves followinga movement of a user's line-of-sight, and thus it is possible toconstantly perform AF at a line-of-sight position corresponding to theuser's line-of-sight. This method is also applicable not only to AF butalso to AE and AWB. This method accumulates information about whichsubject and how long the user gazed the subject during image capturingand stores the information together with images, whereby it becomespossible to propose diverse functions corresponding to the line-of-sightposition and accumulated time information corresponding to the user'sline-of-sight at the playback time. Other than examples of the digitalcamera 100, a certain Head Mount Display (HMD) increases resolutioncentering on a line-of-sight position corresponding to a user'sline-of-sight, and decreases the resolution with increasing distancefrom the line-of-sight position corresponding to the user'sline-of-sight. Certain augmented reality (AR) glasses display detailedinformation for a building or an advertisement existing ahead of auser's line-of-sight.

Eye tracking calibration refers to a calibration step for detecting auser's line-of-sight by using the line-of-sight detection block 160 andmore exactly determining a line-of-sight position corresponding to theuser's line-of-sight. The line-of-sight detection block 160 isconfigured to detect a user's line-of-sight and determine aline-of-sight position corresponding to the line-of-sight, even withoutperforming the calibration. However, since human eye structuresincluding eyelids have individual differences, it is sometimes difficultto determine a line-of-sight position corresponding to a user'sline-of-sight. Performing calibration enables acquiring line-of-sightdata serving as the line-of-sight information specific to the user usingthe digital camera 100. A line-of-sight position corresponding to aline-of-sight input by the user can be more exactly determined bycalculating a calibration value, based on acquired line-of-sight dataspecific to the user. A more accurate calibration value can be acquiredby performing the calibration a plurality number of times. Calibrationvalues applicable to diverse situations can be acquired by performingthe calibration in various situations, for example, under bright naturallight, under a fluorescent light, in a dark place, and in a state whereglasses are put on. In a case where line-of-sight detection is performedin various orientations, as is the case with the digital camera 100according to the present exemplary embodiment, the positional relationbetween the EVF 29 of the digital camera 100 and the eye 161 may bechanged by change in a user's state of looking into the EVF 29. If thepositional relation between the line-of-sight detection block 160 andthe eye 161 is changed during execution of the calibration, theline-of-sight data serving as the line-of-sight information to beacquired largely changes, which results in a deviation from thecalibration value. Therefore, it is desirable to perform the calibrationfor each individual orientation and not desirable that the orientationchanges during the calibration.

Orientations of the digital camera 100, and relative positionalrelations between the line-of-sight detection block 160 and the eye 161are illustrated in FIGS. 6A to 6C. FIG. 6A illustrates four differentcategories of orientations of the digital camera 100. The orientationsof the body of the digital camera 100 are classified into the followingfour categories (positions).

-   -   When the eyepiece 16 is in a range of an area 611 (horizontal        normal position).    -   When the eyepiece 16 is in a range of an area 612 (left vertical        position).    -   When the eyepiece 16 is in a range of an area 613 (right        vertical position).    -   When the eyepiece 16 is positioned in a range of an area 614        (horizontal reverse position)

FIGS. 6B and 6C illustrate positional relations between the infraredemitting diodes 166 and the eye 161 in a state where the user's eye isin proximity to the eyepiece 16. According to the present exemplaryembodiment, the infrared emitting diodes 166 (diodes 166 a to 166 d) aredisposed at four different positions inside the eyepiece 16, asillustrated in FIG. 6B. As described above, the positional relationbetween the line-of-sight detection block 160 and the eye 161, i.e., therelative positional relations between the infrared emitting diodes 166and the eye 161, affects a line-of-sight data to be acquired.

In a case of the digital camera 100, when the eye of the user standingupright is in proximity to the eyepiece 16 of the digital camera 100, achange in the orientation of the digital camera 100 changes the relativepositional relations between the infrared emitting diodes 166(line-of-sight detection block 160) and the eye 161. More specifically,the orientation of the digital camera 100 indicates the relativepositional relation between the line-of-sight detection block 160 andthe eye 161.

FIG. 6B illustrates the positional relations between the infraredemitting diodes 166 (diodes 166 a to 166 d) and the eye 161 in a statewhere the digital camera 100 is held at the horizontal normal position.The diodes 166 a and 166 b emit infrared light from the upper eyelidside of the eye 161, and the diodes 166 c and 166 d emit infrared lightfrom the lower eyelid side of the eye 161.

FIG. 6C illustrates the positional relations between the infraredemitting diodes 166 (diodes 166 a to 166 d) and the eye 161 in a statewhere the digital camera 100 is held at the left vertical position. Thediodes 166 b and 166 d emit infrared light from the upper eyelid side ofthe eye 161, and the diodes 166 a and 166 c emit infrared light from thelower eyelid side of the eye 161.

In a state where the digital camera 100 is held at the right verticalposition (not illustrated), the diodes 166 a and 166 c emit infraredlight from the upper eyelid side of the eye 161, and the diodes 166 band 166 d emit infrared light from the lower eyelid side of the eye 161.In a state where the digital camera 100 is held at the horizontalreverse position (not illustrated), the diodes 166 c and 166 d emitinfrared light from the upper eyelid side of the eye 161, and the diodes166 a and 166 b emit infrared light from the lower eyelid side of theeye 161.

Referring to FIG. 6A, while each of the areas 611 to 614 has an angularrange of 90 degrees (360 degrees are divided into quarters), the angularrange may be different for each area.

In common practice of the calibration, a plurality of gaze points isdisposed at different display positions, line-of-sight data when theuser gases each gaze point is acquired and accumulated, and acalibration value is calculated based on the plurality of line-of-sightdata pieces. In this case, since the plurality of gaze points isdisplayed at different positions, the line-of-sight data of the eyeballcan be acquired for various angles. If the calculated calibration valueis pre-registered as calibration data, the user is able to perform theline-of-sight input with higher accuracy without performing thecalibration each time the user uses the line-of-sight input function.According to the present exemplary embodiment, a calibration valuecalculated based on acquired line-of-sight data and an orientation(orientation information) of the digital camera 100 are stored andregistered in an associated way as calibration data.

In a calibration mode, for example, three or five gaze points are knownto be displayed. Examples of display forms of these gaze points includethe following control method. The plurality of gaze points issequentially displayed one by one. After acquisition of line-of-sightdata for the first gaze point, the first gaze point is hidden and thenthe second gaze point is displayed. After acquisition of all ofline-of-sight data pieces and calculation of a calibration value, thecalibration is completed.

At the calibration in the calibration mode, the user is required to gazeeach gaze point since an unstable line-of-sight is undesired. Forexample, in a control method in which only the first gaze point isdisplayed and then the second gaze point is displayed while the firstone remains displayed, the user may unconsciously move the line-of-sightbetween the first and the second gaze points, which results in anunstable line-of-sight. Therefore, it is more desirable that theplurality of gaze points is sequentially displayed one by one, andline-of-sight data corresponding to each gaze point is acquired.

In control in the calibration mode according to the present exemplaryembodiment, line-of-sight data at a total of five gaze points isacquired, and the gaze points are sequentially displayed one by one.After acquisition of line-of-sight data for one gaze point, the gazepoint is hidden and then the following gaze point is displayed.

The eye-contact detection unit 57 is an eye-contact detection sensorthat detects the state where the eye (object) 161 is coming closer to(coming into contact with) the eyepiece 16 of the finder (eye-proximitystate) and the state where the eye 161 is being detached from (comingout of contact with) the eyepiece 16 of the finder (eye out-of-proximitystate). The system control unit 50 turns display of the display unit 28and the EVF 29 ON (display state) or OFF (not display state) accordingto the state detected by the eye-contact detection unit 57. Morespecifically, at least in a case where the digital camera 100 is in theimage capturing standby state and in a case where automatic changeoversetting is made for the display destination of a live view imagecaptured by the imaging unit 22, the following display control isperformed. In the eye out-of-proximity state, the display unit 28 is setas the display destination, i.e., the display of the display unit 28 isturned ON and the display of the EVF 29 is turned OFF. In theeye-proximity state, on the other hand, the EVF 29 is set as the displaydestination, i.e., the display of the EVF 29 is turned ON and thedisplay of the display unit 28 is turned OFF.

The eye-contact detection unit 57 can use, for example, an infraredproximity sensor for detecting a state where a certain object is comingcloser to the eyepiece 16 of the finder including the EVF 29. In a casewhere an object comes close to the eyepiece 16, infrared light emittedfrom the light projecting portion (not illustrated) of the eye-contactdetection unit 57 is reflected and then received by the light receivingportion (not illustrated) of the infrared light proximity sensor. Theeye-contact detection unit 57 can also determine a distance (eye-contactdistance) from the eyepiece 16 to the object, based on the amount of thereceived infrared light. In this way, the eye-contact detection unit 57performs eye-contact detection for detecting the eye-contact distancefrom the object to the eyepiece 16.

According to the present exemplary embodiment, the light projectingportion and the light receiving portion of the eye-contact detectionunit 57 are devices different from the infrared emitting diodes 166 andthe line-of-sight detection sensor 164, respectively. However, theinfrared emitting diodes 166 may also serve as the light projectingportion of the eye-contact detection unit 57, and the line-of-sightdetection sensor 164 may serve as the light receiving portion.

In a case where an object coming closer to the eyepiece 16 at apredetermined distance or less is detected in the eye out-of-proximitystate (non-proximity state), the eye-contact detection unit 57determines the eye-proximity state. In a case where an object in theeye-proximity state (proximity state) has been detached from theeyepiece 16 by a predetermined distance or longer, the eye-contactdetection unit 57 determines the eye out-of-proximity state. A thresholdvalue for detecting the eye-proximity state and a threshold value fordetecting the eye out-of-proximity state may be different, for example,by providing a hysteresis. Once the eye-proximity state is detected, theeye-proximity state continues until the eye out-of-proximity state isdetected. Once the eye out-of-proximity state is detected, the eyeout-of-proximity state continues until the eye-proximity state isdetected. The infrared proximity sensor is an example of the eye-contactdetection unit 57. Other sensors capable of detecting approach of an eyeor object (with which the eye-proximity state can be determined) may beemployed as the eye-contact detection unit 57.

The system control unit 50 can detect the following operations andstates, based on the output from the line-of-sight detection block 160.

-   -   A state where a new line-of-sight of the user's eye in proximity        to the eyepiece 16 is input (detected), i.e., the user starts        the line-of-sight input.    -   A state where the user's eye in proximity to the eyepiece 16 is        being subjected to the line-of-sight input.    -   A state where the user's eye in proximity to the eyepiece 16 is        gazing a gaze point.    -   A state where the user's eye in proximity to the eyepiece 16        shifts the line-of-sight, i.e., the user ends the line-of-sight        input.    -   A state where the user's eye in proximity to the eyepiece 16 is        not performing the line-of-sight input.

“Gaze” described above refers to a state where the line-of-sightposition corresponding to the user's line-of-sight does not exceed (isequal to or less than) a predetermined moving amount within apredetermined time period.

The present exemplary embodiment will be described below centering onprocessing for performing the calibration using the user's line-of-sightwith the digital camera 100 as an example of an electronic apparatus.

FIG. 3 is a control flowchart illustrating processing that is executedwhen the user performs eye tracking calibration. This control processingis implemented when the system control unit 50 loads a program stored inthe nonvolatile memory 56 into the system memory 52 and then executesthe program.

The flowchart in FIG. 3 is started when the user performs an operationfor starting the calibration on a setting menu screen in a state wherethe digital camera 100 is activated and the user is looking into thefinder, i.e., the user's eye is in proximity to the eyepiece 16. FIG. 7illustrates an example of a setting menu screen for the line-of-sightinput function. The line-of-sight input setting as an item 701 in FIG. 7indicates a state where an option 701 a is selected, i.e., theline-of-sight input is set to ON. The flowchart in FIG. 3 is started ina case where an execution instruction is issued for an item 702, i.e.,in a case where a display item 702 a is selected and determined.Referring to FIG. 7 , “Not Registered” is displayed for the display item702 a, which indicates that the user has not performed the calibration.In a case where the user has performed the calibration even at leastonce, message “Registered” is displayed for the display item 702 a. Thecalibration can be performed as described above when whichever of themessages is displayed for the display item 702 a.

In step S301, the system control unit 50 displays a screen for startingthe calibration in the EVF 29. FIG. 5A illustrates an example display ofa screen for starting the calibration. The example illustrated in FIG.5A is displayed in the EVF 29 in a case where the user's eye is inproximity to on the eyepiece 16 and the orientation detection unit 55detects that the orientation of the digital camera 100 is the horizontalnormal position. At the time of user's line-of-sight detection includingthe calibration, some adjustments is performed by the user to make iteasier for the line-of-sight detection block 160 to detect the user'sline-of-sight. Therefore, at the start of the calibration, a dialog 501in FIG. 5A is displayed. Even if the user has not used the line-of-sightinput function, displaying a dialog like the dialog 501 makes it easierfor the user to grasp tips on the line-of-sight input as well as thecalibration. Messages of the dialog 501 are not limited thereto, and thedialog 501 in FIG. 5A may not be displayed. The system control unit 50may start processing from step S302, skipping step S301, in response tothe issuance of an instruction for starting the calibration.

In step S302, the system control unit 50 determines whether adetermination operation is performed. In a case where the determinationoperation is performed (YES in step S302), the processing proceeds tostep S303. On the other hand, in a case where the determinationoperation is not performed (NO in step S302), the processing proceeds tostep S306. The determination operation according to the presentexemplary embodiment is pressing of the M-Fn button 77 as illustrated bya display item 511 in FIG. 5A. The determination operation is acondition as a declaration of intent by the user to proceed to thefollowing step (screen), and is not limited to pressing of the M-Fnbutton 77.

In step S303, the system control unit 50 starts processing for acquiringcalibration data. The processing for acquiring the calibration data willbe described in detail below with reference to the control flowcharts inFIGS. 4A and 4B.

In step S304, the system control unit 50 displays an end message uponcompletion of the calibration, instead of displaying the gaze points.More specifically, the system control unit 50 displays a dialog 503 andoptions 512 and 513 as illustrated in FIG. 5G.

In step S305, the system control unit 50 determines whether aninstruction for performing the calibration again is issued. In a casewhere the instruction is issued (YES in step S305), the processingreturns to step S400. On the other hand, in a case where the instructionis not issued (NO in step S305), the processing exits the controlflowcharts in FIGS. 4A and 4B. The instruction for performing thecalibration again refers to selection of the option 513 illustrated inFIG. 5G. In a case where the user selects the option 513, the systemcontrol unit 50 performs the calibration again from the beginning. In acase where the user selects the option 512, on the other hand, thesystem control unit 50 changes the calibration instruction screen to ascreen displayed before the user has started calibration, for example,the setting menu screen.

Performing the calibration a plurality of times allows acquisition ofcalibration data having higher accuracy. Displaying the screen in FIG.5G allows informing the user of acquisition of calibration data havinghigher accuracy by performing the calibration a plurality of times,whereby an effect of promoting the user to perform the calibration aplurality of times can be acquired. Upon completion of line-of-sightdata acquisition at the five gaze points, the system control unit 50 mayend the processing for acquiring calibration data without displaying thedialog 503 illustrated in FIG. 5G. More specifically, the processing mayskip steps S304 and S305 and exit the control flowchart in FIG. 3directly from step S303.

In step S306, the system control unit 50 determines whether an operationother than the determination operation is performed. In a case where anoperation other than the determination operation is performed, i.e., anoperation is performed on an operation member other than the M-Fn button77 described in step S302 (YES in step S306), the system control unit 50ends the calibration processing. On the other hand, in a case where suchan operation is not performed (NO in step S306), the processing returnsto step S302. In a case where an operation is performed on an operationmember other than the M-Fn button 77, the system control unit 50performs processing corresponding to an operated member. For example, ina case where the user presses the playback button 79, the system controlunit 50 transits to playback mode processing to play back an imagecaptured or stored in the recording medium 200. In a case where the userpresses the shutter button 61, the system control unit 50 transits tothe image capturing standby state as image capturing mode processing,and performs image capturing preparation processing, such as the AF, AE,and AWB processing. When an image capturing instruction is issued, thesystem control unit 50 performs a series of image capturing processingup to recording of a captured image in the recording medium 200 as animage file.

FIGS. 4A and 4B are control flowcharts illustrating processing foracquiring the calibration data explained in step S303 in FIG. 3 . Whenthe determination operation is performed (YES in step S302), the systemcontrol unit 50 starts the processing for acquiring the calibrationdata.

In step S400, the system control unit 50 initializes flag information F(F=0) and stores the flag information F in the system memory 52. Theflag information F indicates whether to permit an orientation changeduring execution of the calibration. In a case where the flaginformation F is 0, the system control unit 50 notifies the user thatthe orientation has changed during execution of the calibration. Theflag information F will be described in detail below.

In step S401, the system control unit 50 sets a variable X to 1 (X=1)and stores the variable X in the system memory 52. According to thepresent exemplary embodiment, the variable X denotes the number of gazepoints to be displayed in the EVF 29.

In step S402, the system control unit 50 refers to the nonvolatilememory 56 and displays the X-th gaze point as calibration display in theEVF 29. A plurality of calibration images associated with an orientationof the digital camera 100 and display order (X) is prestored in thenonvolatile memory 56. In a case where the digital camera 100 is at ahorizontal position (normal position), the system control unit 50displays a gaze point 521 in FIG. 5B (X=1), a gaze point 522 in FIG. 5C(X=2), a gaze point 523 in FIG. 5D (X=3), a gaze point 524 in FIG. 5E(X=4), and a gaze point 525 in FIG. 5F (X=5) in the EVF 29. Morespecifically, these gaze points are displayed at the center, left,right, top, and bottom positions in this order when viewed from theuser. This display order is determined in consideration of stability ofthe user's line-of-sight. As described above, instead of sequentiallydisplaying the gaze points one by one as illustrated in FIGS. 5A to 5H,the system control unit 50 may display all of the five gaze points inthe EVF 29 and then sequentially hide the five gaze points one by one.More specifically, the system control unit 50 may sequentially hide thegaze points from the one determined to have been gazed by the user (theline-of-sight data has been acquired). Then, the system control unit 50may ends the control flowcharts in FIGS. 4A and 4B upon completion ofthe line-of-sight data acquisition for the last gaze point.

For example, if a gaze point is displayed at the bottom of the EVF 29and then displayed at the top, the user needs to move the line-of-sightfrom the bottom to upward. This may possibly degrade stability of theline-of-sight because of the characteristics of the human eye. For thisreason, in a state where the eyepiece 16 is on the left side and thegrip portion 90 is at the top (left vertical position), the gaze pointsare displayed at positions illustrated in FIGS. 5B, 5E, 5F, 5D, and 5Cin this order so that their relative positions in the EVF 29 change inorder of the center, left, right, top, and bottom positions. In a statewhere the eyepiece 16 is on the right side and the grip portion 90 is atthe bottom (right vertical position), the gaze points are displayed atpositions illustrated in FIGS. 5B, 5F, 5E, 5C, and 5D in this order.

More specifically, the gaze points are relatively displayed at thecenter, left, right, top, and bottom positions in this order when viewedfrom user, regardless of the orientation of the digital camera 100.While, in the display form according to the present exemplaryembodiment, the system control unit 50 displays the gaze points in theabove-described order, i.e., the display order changes according to theorientation of the digital camera 100, embodiments of the presentinvention are not limited thereto. The number of gaze points is notlimited thereto but may be four (top, bottom, left, and right), three(top, lower left, and lower right), or other numbers.

In step S403, like step S302, the system control unit 50 determineswhether the determination operation is performed. In a case where thedetermination operation is performed (YES in step S403), the processingproceeds to step S404. On the other hand, in a case where thedetermination operation is not performed (NO in step S403), theprocessing proceeds to step S427.

In step S404, the system control unit 50 determines orientationinformation P(X) of when the determination operation is performed instep S403, as the orientation of the digital camera 100. As describedabove, the orientation of the digital camera 100 is constantly beingdetected by the orientation detection unit 55. However, the systemcontrol unit 50 determines the orientation of when the determinationoperation as a declaration of intent by the user is performed, as theorientation of the digital camera 100.

In step S405, the system control unit 50 determines whether the variableX is 1. In a case where the variable X is 1 (YES in step S405), theprocessing proceeds to step S406. On the other hand, in a case where thevariable X is not 1 (NO in step S405), the processing proceeds to stepS408.

In step S406, the system control unit 50 determines whether theorientation information P(X) for the digital camera 100 determined instep S404 is an orientation that is registerable as calibration data. Ina case where the orientation information P(X) is determined to be aregisterable orientation (YES in step S406), the processing proceeds tostep S407. On the other hand, in a case where the orientationinformation P(X) is determined to be an unregisterable orientation (NOin step S406), the processing proceeds to step S416.

The orientations that can be registered as calibration data include thehorizontal normal position, the left vertical position, and the rightvertical position out of the four different orientations illustrated inFIG. 6A. In a case where the orientation of the digital camera 100 isthe horizontal reverse position, the orientation cannot be registered ascalibration data. This is because image capturing using theline-of-sight input is considered to be unlikely performed in a statewhere the eyepiece 16 is in the range of the area 614. However, even thehorizontal reverse position may be registered as calibration data.

In step S407, the system control unit 50 stores the orientation of thedigital camera 100 determined in step S404 as orientation informationP(X) in the nonvolatile memory 56. According to the present exemplaryembodiment, the system control unit 50 stores in the nonvolatile memory56 only the orientation of when the determination operation is performedin a state where the first gaze point is displayed.

In a case where the system control unit 50 determines that the variableX is not 1 (NO in step S405), the processing proceeds to step S408. Instep S408, the system control unit 50 determines whether the orientationinformation P(X) of the digital camera 100 is an orientation P(1) ofwhen the determination operation is performed in a state where the firstgaze point is displayed. In a case where P(X) is P(1) (YES in stepS408), the processing proceeds to step S409. On the other hand, in acase where P(X) is not P(1) (NO in step S408), the processing proceedsto step S421.

In step S409, the system control unit 50 determines line-of-sight dataE(X) when the X-th gaze point is displayed. According to the presentexemplary embodiment, the user's line-of-sight is constantly beingdetected by the line-of-sight detection block 160 during thecalibration. In step S409, the system control unit 50 determinesinformation about the line-of-sight acquired at a timing of when thedetermination operation is input in step S403, as line-of-sight dataE(X). The reason why the system control unit 50 determines theline-of-sight data acquired at the timing of when the determinationoperation is performed is to acquire more reliable line-of-sight data.The line-of-sight data may be acquired at a timing of when the systemcontrol unit 50 determines that a gaze is detected by the line-of-sightdetection block 160, even in a case where the determination operation isnot performed. In a case where a gaze is used for the determinationoperation at the time of calibration data acquisition as in the controlflowcharts in FIGS. 4A and 4B, the system control unit 50 may setanother condition, which is different from the gaze determinationcondition, for the time other than calibration data acquisition. Forexample, at the time other than calibration data acquisition, the systemcontrol unit 50 determines that a gaze is detected in a case where themoving amount of the user's line-of-sight is equal to or less than apredetermined threshold value during 120 milliseconds. On the otherhand, at the time of calibration data acquisition, the system controlunit 50 determines that a gaze is detected in a case where the movingamount of the user's line-of-sight is equal to or less than thethreshold value during three seconds. The predetermined threshold valueof the moving amount of the line-of-sight may be decreased while thecriterion time for gaze determination is maintained to 120 milliseconds,or the predetermined threshold value may be decreased with prolongedcriterion time.

In step S410, the system control unit 50 determines whether theline-of-sight data E(X) determined in step S409 can be used asline-of-sight data (OK) or not (NG). In a case where the line-of-sightdata E(X) can be used as line-of-sight data (OK) (YES in step S410), theprocessing proceeds to step S411. On the other hand, in a case where theline-of-sight data E(X) cannot be used as line-of-sight data (NG) (NO instep S410), the processing proceeds to step S416. Causes of NGdetermination of the line-of-sight data include natural light that isreflected by the eye or reflected by the lenses of glasses at the timingof when the determination operation is performed in step S403. There isalso a case where the line-of-sight data E(X) suitable for calculatingthe calibration value in step S413 (described below) cannot be acquiredbecause the user blinks a number of times.

In step S411, the system control unit 50 inputs X+1 to the variable Xand stores the variable X in the system memory 52.

In step S412, the system control unit 50 refers to the system memory 52to determines whether X is larger than a threshold value Xth (X>Xth). Ina case where X>Xth (YES in step S412), the processing proceeds to stepS413. On the other hand, in a case where X≤Xth (NO in step S412), theprocessing proceeds to step S414. The threshold value Xth indicates thenumber of gaze points for acquiring the line-of-sight data at thecalibration. According to the present exemplary embodiment, Xth is 5since five gaze points are displayed.

In step S413, the system control unit 50 calculates the calibrationvalue, based on all of the line-of-sight data pieces E(X) acquired untilthe processing reaches this step. Then, the processing proceeds to stepS415.

In step S414, the system control unit 50 hides the gaze points to bedisplayed in the EVF 29. Then, the processing returns to step S402. Asdescribed above, in control in the calibration mode according to thepresent exemplary embodiment, the gaze points as calibration display aresequentially displayed one by one, and only one gaze is displayed in theEVF 29 at a time. Therefore, the gaze points are hidden in this step.

In step S415, the system control unit 50 combines P(1) stored in stepS407 and the calibration value calculated in step S413 as calibrationdata, and stores these data pieces in the nonvolatile memory 56 in anassociated way. As described above, as the relative positional relationbetween the line-of-sight detection block 160 and the eye 161 changes,the position of the infrared light emitted from the infrared emittingdiodes 166 and reflected by the eye 161 changes, and consequently theinfrared light detected by the line-of-sight detection block 160 alsochanges. Thus, by storing the orientation of the line-of-sight detectionblock 160, i.e., by storing the orientation of the digital camera 100and the user's line-of-sight data in an associated way, the user is ableto perform position specification by the line-of-sight input havinghigher accuracy in a case where the user executes a function based onthe line-of-sight input.

In a case where the orientation information P(1) is determined to be anunregisterable orientation (NO in step S406) or in a case where theline-of-sight data E(X) is determined to be NG (NO in step S410), thenin step S416, the system control unit 50 displays a warning in the EVF29. In a case where the orientation information P(1) is determined to bean unregisterable orientation (NO in step S406), a warning “OperateCamera in Horizontal Position or Vertical Position” is displayed. In acase where the line-of-sight data E(X) is determined to be NG (NO instep S410), a warning according to the cause of NG line-of-sight dataE(X) is displayed. Example causes of NG line-of-sight data include anumber of blinks, natural light reflected by the eye, and glassescovered by ghost. In these cases, the system control unit 50 displays“Keep Opening Both Eyes”, “Do Not Blink When Pressing M-Fn Button”, and“Raise Positions Of Glasses Before Bringing The Eye Close To Finder”,respectively. The system control unit 50 also displays “Bring The EyeClose To Finder” and “Perform Calibration With Another RegistrationNumber When Another Person Uses Camera Or When Glasses Are Put On OrOff”. Displaying such warnings allows prompting the user to improve thecamera orientation and eye conditions so that the line-of-sight data canbe acquired. The system control unit 50 displays a warning in the EVF 29while displaying the gaze points. The system control unit 50 may oncehide the gaze points and then display a warning.

In step S417, like step S403, the system control unit 50 determineswhether the determination operation is performed. In a case where thedetermination operation is performed (YES in step S417), the processingproceeds to step S418. On the other hand, in a case where thedetermination operation is not performed (NO in step S417), theprocessing proceeds to step S419.

In step S418, the system control unit 50 hides the warning displayed instep S416. Then, the processing returns to step S402. The system controlunit 50 may hide the warning (a dialog 504 in FIG. 5H) displayed in stepS416 when a predetermined time period has elapsed since the display isstarted, not based on the determination whether the determinationoperation is performed in step S417.

In step S419, like step S304, the system control unit 50 determineswhether an operation other than the determination operation isperformed. In a case where an operation other than a determinationoperation is performed (YES in step S419), the processing proceeds tostep S420. On the other hand, in a case where an operation other thanthe determination operation is not performed (NO in step S419), theprocessing returns to step S417.

In step S420, the system control unit 50 discard the data acquired inthe flowcharts in FIGS. 4A and 4B, i.e., the orientation informationP(1) and the line-of-sight data E(X) of the digital camera 100 stored inthe system memory 52. Then, the processing exits the control flowchartsin FIG. 3 and FIGS. 4A and 4B. In step S420, since an instruction forending the data acquisition processing is received before completion ofthe line-of-sight data acquisition for all of the five gaze points, thesystem control unit 50 discards all of the acquired data. Then, theprocessing exits the control flowcharts in FIGS. 4A and 4B. In otherwords, since an instruction for ending the data acquisition processingis received before completion of a series of processing for calculatingthe calibration value based on the line-of-sight data for all of thefive gaze points and storing the calculation value in association withthe orientation information of the camera, the system control unit 50discards all of the acquired data. The above-described control makes itpossible to prevent the calibration value from being calculatedregardless of the insufficient number of line-of-sight data pieces asline-of-sight information for calculating the calibration value. Thisenables avoiding the acquisition and storage of the low-accuracycalibration value, i.e., calibration data.

In a case where the orientation information P(X) for the X-th gaze pointis determined to be different from the orientation information P(1) forthe first gaze point (NO in step S408), the processing proceeds to stepS421. In step S421, the system control unit 50 determines whether theflag information F is 1. In a case where the flag information F is 1(YES in step S421), the processing proceeds to step S409. On the otherhand, in a case where the flag information F is not 1 (the flaginformation F is 0) (NO in step S421), the processing proceeds to stepS422.

In a case where the orientation information P(X) for the X-th gaze pointis determined to be different from the orientation information P(1) forthe first gaze point (NO in step S408) and in a case where the flaginformation F is determined to be not 1 (NO in step S421), then in stepS422, the system control unit 50 displays a warning. An example displayis illustrated in FIG. 5H. The system control unit 50 displays a warningto notify the user that the current orientation is different from theorientation of the line-of-sight data acquired at the first gaze point.By displaying a warning such as the dialog 504 illustrated in FIG. 5H,the system control unit 50 is able to prompt the user to restore theorientation to the orientation of the line-of-sight data acquired at thefirst gaze point. According to the present exemplary embodiment, thesystem control unit 50 displays a warning while maintaining the displayof a gaze point, as illustrated in FIG. 5H. Performing control in thisway makes it unnecessary for the user to move the line-of-sight tosearch for a gaze point at a timing of when the dialog 504 is hidden.However, the system control unit 50 may perform control so that none ofthe gaze point and the warning is displayed in the EVF 29.

In step S423, the system control unit 50 determines whether theorientation of the digital camera 100 is restored. The system controlunit 50 monitors output data of the orientation detection unit 55 todetermine whether the same orientation information as P(1) is acquired.In a case where the same orientation information as P(1) is acquired(YES in step S423), the processing proceeds to step S426. On the otherhand, In a case where the same orientation information as P(1) is notacquired (NO in step S423), the processing proceeds to step S424.

In step S424, like step S403, the system control unit 50 determineswhether the determination operation is performed. In a case where thedetermination operation is performed, i.e., in a case where the M-Fnbutton 77 as a display item 514 in FIG. 5H is pressed (YES in stepS424), the processing proceeds to step S425. On the other hand, in acase where the determination operation is not performed (NO in stepS424), the processing returns to step S423.

In step S425, the system control unit 50 sets the flag information F to1.

In a case where the determination operation is performed (YES in stepS424), it can be considered that the user ignores the warning displayand wants to continue the calibration with an orientation different fromthe orientation information P(1). Thus, in the following processing, thesystem control unit 50 can continue the acquisition of the line-of-sightdata E(X) without repeating the warning display even in a case where theorientation information P(X) acquired in step S403 is different from theorientation information P(1) for the first gaze point. For example, in acase where P(1), P(2), and P(3) indicate the normal position, theorientation is changed to the left vertical position based on P(4), andthe determination operation is performed in step S424 with theorientation set to the left vertical position, the system control unit50 hides the warning, and the processing returns to step S402. In stepS402, the system control unit 50 displays the fourth gaze point (X=4).Then, even in a case where the user performs the determination operationin step S403 with the orientation of the body of the digital camera 100set to the left vertical position, the processing proceeds to step S409from step S404 followed by NO in step S405 and YES in steps S408. Instep S409, the system control unit 50 accumulates the line-of-sight dataE(X) as the line-of-sight data corresponding to the X-th gaze point.

In this control, the line-of-sight data acquired and accumulated isdegraded in accuracy for the orientation of the digital camera 100.However, this control enables reducing the burden of returning to thefirst gaze point even after completion of the calibration for up to thefourth gaze point. If the gaze point display after the orientation ofthe camera is changed is returned to the first gaze point (the gazepoint displayed first), the user is highly likely to feel burdensome tostart the calibration from the beginning. For this reason, the user maynot use the line-of-sight input function or may use the line-of-sightinput without performing the calibration. If the user uses theline-of-sight input without performing the calibration, a differenceoccurs between the user's line-of-sight and the line-of-sight positioncorresponding to the line-of-sight detected by the digital camera 100,causing the degradation of usability. As a result, an issue that theuser does not use the line-of-sight input may arise. Therefore, whilepromoting the use of the line-of-sight input function, the systemcontrol unit 50 is able to prompt the user to perform the calibrationfor using the line-of-sight input having higher accuracy without feelingburdensome.

The system control unit 50 may repeat the determination processing instep S423 until the orientation of the digital camera 100 becomesequivalent to P(1) even in a case where the determination operation isperformed. In this case, since the line-of-sight data in a case wherethe orientation of the body of the digital camera 100 is approximatelyequivalent to P(1) can be collected, calibration data having higheraccuracy can be acquired and registered. This enables reducing thedifference between the position currently being gazed by the user andthe line-of-sight position corresponding to the line-of-sight calculatedby the calibration.

In step S426, the system control unit 50 hides the warning displayed instep S422. Then, the processing returns to step S402.

After the image indicating the X-th gaze point is displayed, in a casewhere the system control unit 50 determines that the determinationoperation is not performed (NO in step S403), the processing proceeds tostep S427. In step S427, like step S304, the system control unit 50determines whether an operation other than the determination operationis performed. In a case where an operation other than the determinationoperation is performed (YES in step S427), the processing exits thecontrol flowcharts in FIG. 3 and FIGS. 4A and 4B. On the other hand, ina case where an operation other than the determination operation is notperformed (NO in step S427), the processing returns to step S403.

While, in the present exemplary embodiment, the determination operationis pressing of the M-Fn button 77, the determination operation is notlimited to the depression of the M-Fn button 77. This is because thedetermination operation is a condition as a declaration of intent by theuser to proceed to the following step (screen), as described above. Anoperation member having other functions that can be assigned thedetermination function is applicable. In a case where other operationmembers are to be used, the operability will further improve with anoperation member disposed at a position where the user can operate theoperation member while keeping the eye in proximity to the eyepiece 16and gripping the grip portion 90 with the right hand. For example, theuser is able to press an operation member disposed on the lens barrel asthe exterior of the lens unit 150 with the left hand while keeping theeye in proximity to the eyepiece 16 and gripping the grip portion 90with the right hand. The determination operation may be determined tohave been performed when the user gazes a displayed gaze point, not whenan operation member is pressed. More specifically, in addition topressing of the M-Fn button 77, the determination operation may bedetermined to have been performed by the user in case where the user'sline-of-sight gazing the gaze point displayed in the EVF 29 does notmove to exceed a predetermined moving amount (i.e., the moving amountremains equal to or less than a predetermined threshold value) during apredetermined time period. In this control, the line-of-sight data isnot determined until the user's line-of-sight becomes stable (until theuser's line-of-sight is determined to be stable to such an extent thatthe line-of-sight data can be acquired). Thus, according to the presentexemplary embodiment, there is a low possibility that the line-of-sightdata E(X) is determined to be NG in step S410 in FIG. 4A.

The warning displayed in steps S416 and S422 may be an icon-basedmessage instead of a text-based message. More specifically, the warningmay be a change of the display format or display color of the EVF 29,blinking of the EVF 29, or a combination of both as long as a warningcan be notified to the user. For example, the dialog 504 in FIG. 5H mayblink or the background color of the EVF 29 may be differentiated fromthe background color for the gaze point display.

As described above, in a case where the orientation of the body changesduring execution of the calibration, the present exemplary embodimentdisplays a warning, prompts the user to restore the orientation of theline-of-sight data determined for the first gaze point, and receives noline-of-sight data after the orientation change until an instruction isissued by the user. This control makes it possible to reduce thedegradation of the line-of-sight data accuracy, and generate and acquirecalibration data having higher accuracy. In a case where thedetermination operation is performed by the user, the system controlunit 50 continuously display the gaze point displayed before theorientation change without restoring the first gaze point even in a casewhere the orientation changes. This enables reducing the possibilitythat the user finds it burdensome to perform the calibration.Accordingly, this control enables acquiring data in such a way that theuser does not feel it burdensome to perform the calibration, whilereducing the degradation of calibration data accuracy.

The above-described various control described to be performed by thesystem control unit 50 may be performed by one hardware component, orthe entire apparatus may be controlled by a plurality of hardwarecomponents (for example, a plurality of processors and circuits) whichshare processing.

While various embodiments of the present disclosure have specificallybeen described based on certain exemplary embodiments, the presentinvention is not limited to these specific exemplary embodiments.Diverse embodiments not departing from the spirit and scope of thepresent invention are also included in the scope of the presentdisclosure. While the gaze points that are used in the calibration havea square shape, the gaze points may have any shape as long as the useris able to gaze the gaze points. The gaze points may be animationsinstead of stationary images. While the background of the gaze pointdisplay is a solid color, the background may change in color, displaycolor gradations, or display a playback image or LV image. For example,to eliminate the need of performing the calibration a number of times atbright and dark places, the system control unit 50 may change thebackground color and luminance each time a plurality of gaze points isdisplayed. This enables the user to perform the calibration undervarious situations with a single calibration operation.

While the present exemplary embodiment has been described abovecentering on an example case of the digital camera 100 having the EVF29, the present invention is not limited thereto. Embodiments of thepresent disclosure are also applicable to electronic apparatusesincluding an acquisition unit configured to acquire information aboutthe user's line-of-sight. More specifically, embodiments of the presentdisclosure are applicable to apparatuses capable of displaying a gazepoint for calibration on a display unit and detecting the user'sline-of-sight gazing the display unit. The above-described exemplaryembodiments may be suitably combined. According to the present exemplaryembodiment, the eyepiece 16 having the EVF 29 and the line-of-sightdetection block 160 are integrally configured. However, the presentexemplary embodiment is also applicable to apparatuses that include adisplay unit and a line-of-sight detection unit as separate units andare configured to detect the user's line-of-sight by connecting theseunits. More specifically, embodiments of the present disclosure areapplicable to personal computers, personal digital assistants (PDAs),mobile phone terminals including smart phones, tablet personalcomputers, portable image viewers, printer apparatuses having a display,digital photo frames, music players, game machines, electronic bookreaders, and wearable devices, such as head mount displays.

For example, in a case where an external line-of-sight detection deviceis connected to a personal computer, control according to the presentexemplary embodiment is performed not with the orientation of thepersonal computer but with the orientation of the line-of-sightdetection device. In the following case, acquisition of the user'sline-of-sight data for the calibration is started in a state that theline-of-sight detection device is horizontally disposed, and isperformed. In a case where the orientation of the line-of-sightdetection device is changed from a horizontal position to a verticalposition during the calibration, the present exemplary embodimentneither acquire nor store the user's line-of-sight data after theorientation of the line-of-sight detection device is changed to thevertical position.

Embodiments of the present disclosure are applicable not only to thebody of an imaging apparatus but also to a control apparatus thatcommunicates with the imaging apparatus (including a network camera) viawire lined or wireless communication to remotely control the imagingapparatus. Examples of apparatuses that remotely control an imagingapparatus include smart phones, tablet PCs, and desktop PCs. The controlapparatus is able to remotely control the imaging apparatus bytransmitting commands for instructing the imaging apparatus to performvarious operations and settings to the imaging apparatus, based onoperations and processing performed on the control apparatus side. Thecontrol apparatus may also be able to receive, via wire lined orwireless communication, a live view image captured by the imagingapparatus and display the live view image.

The above-described various control described to be performed by thesystem control unit 50 may be performed by one hardware component, orthe entire apparatus may be controlled by a plurality of hardwarecomponents (for example, a plurality of processors and circuits) whichshare processing.

Other Exemplary Embodiments

Embodiments of the present disclosure can also be implemented byperforming the following processing. More specifically, software(program) for implementing the functions of the above-describedexemplary embodiments is supplied to a system or apparatus via a networkor various types of storage media, and a computer (or central processingunit (CPU) or micro processing unit (MPU)) of the system or apparatusreads and executes the program code. In this case, the program and thestorage medium storing the program are included in the presentinvention.

Various embodiments of the present disclosure make it possible togenerate calibration data having higher accuracy.

OTHER EMBODIMENTS

Embodiment(s) of the present disclosure can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While exemplary embodiments have been described, it is to be understoodthat the invention is not limited to the disclosed exemplaryembodiments. The scope of the following claims is to be accorded thebroadest interpretation so as to encompass all such modifications andequivalent structures and functions.

This application claims the benefit of Japanese Patent Application No.2020-121212, filed Jul. 15, 2020, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An electronic apparatus comprising: a detectionunit configured to detect an orientation of a body including anacquisition unit; and at least one memory and at least one processorwhich function as: an acquisition unit configured to acquire informationabout a user's line-of-sight to a screen; and a control unit configuredto perform control to execute an eye tracking calibration, based on theacquired information about the user's line-of-sight, wherein the controlunit performs control to acquire first orientation informationindicating a first orientation of the body and a first line-of-sightinformation about the user's line-of-sight, of a case where a first eyetracking calibration display is displayed on the screen; acquire secondorientation information indicating a second orientation of the body anda second line-of-sight information about the user's line-of-sight, of acase where a second eye tracking calibration display is displayed on thescreen; and cause, in a case where the second orientation information isdifferent from the first orientation information, the screen to displaya predetermined display that prompts the user to change the orientationof the body to the first orientation.
 2. The electronic apparatusaccording to claim 1, wherein the predetermined display is displayed onthe screen together with the second eye tracking calibration display. 3.The electronic apparatus according to claim 1, wherein, in a case wherethe second orientation information is different from the firstorientation information, the control unit does not use the secondline-of-sight information for the eye tracking calibration.
 4. Theelectronic apparatus according to claim 1, wherein, in a case where apredetermined condition is satisfied without a change in the orientationof the body from the second orientation to the first orientation afterthe predetermined display is displayed on the screen, the control unituses the line-of-sight information for the eye tracking calibration. 5.The electronic apparatus according to claim 1, wherein, after displayingeither one of an eye tracking calibration display including the firsteye tracking calibration display and the second eye tracking calibrationdisplay on the screen, the control unit acquires information about theorientation of the body and the user's line-of-sight in response to apredetermined condition being satisfied.
 6. The electronic apparatusaccording to claim 4, wherein the predetermined condition is either anoperation on an operation member or a state where a moving amount of aposition where the user's line-of-sight is detected in a predeterminedtime period is equal to or less than a predetermined threshold value. 7.The electronic apparatus according to claim 1, wherein each of the firsteye tracking calibration display and the second eye tracking calibrationdisplay is a display including a gaze point that is displayed on thescreen.
 8. The electronic apparatus according to claim 7, wherein thefirst eye tracking calibration display is a display where the gaze pointis disposed at a first position on the screen, and the second eyetracking calibration display is a display where the gaze point isdisposed at a second position different from the first position on thescreen.
 9. The electronic apparatus according to claim 7, wherein thecontrol unit stores, in an associated way, the orientation of the bodyacquired at the time of acquisition of the information about the user'sline-of-sight of when a first gaze point is displayed, and the eyetracking calibration value of the line-of-sight calculated based on theinformation about the user's line-of-sight acquired during execution ofthe eye tracking calibration.
 10. The electronic apparatus according toclaim 1, wherein, in a case where a process of the eye trackingcalibration is ended before completion, the control unit discards theinformation about the user's line-of-sight and the orientation of thebody that are acquired by the acquisition unit and the detection unit,respectively, during execution of the eye tracking calibration.
 11. Theelectronic apparatus according to claim 1, wherein, in a case where theorientation of the body changes from the second orientation to the firstorientation after the predetermined display is displayed on the screen,the control unit hides the predetermined display.
 12. The electronicapparatus according to claim 1, further comprising: an image capturingunit configured to capture a subject image; a finder; and aninside-finder display unit, wherein the control unit acquiresinformation about the user's line-of-sight with respect to the screendisplayed on the inside-finder display unit.
 13. A method forcontrolling an electronic apparatus, the method comprising: acquiringinformation about a user's line-of-sight to a screen; detecting anorientation of a body; and performing control to execute an eye trackingcalibration, based on the acquired information about the user'sline-of-sight, wherein the control is performed to acquire firstorientation information indicating a first orientation of the body andfirst line-of-sight information about the user's line-of-sight, of acase where a first eye tracking calibration display is displayed on thescreen; acquire second orientation information indicating a secondorientation of the body different from the first orientation and secondline-of-sight information about the user's line-of-sight, of a casewhere a second eye tracking calibration display is made on the screen;and cause, in a case where the second orientation information isdifferent from the first orientation information, the screen to displaya predetermined display for prompting the user to change the orientationof the body to the first orientation.
 14. A non-transitorycomputer-readable recording medium storing a program for executing amethod for controlling an electronic apparatus, the method forcontrolling an electronic apparatus comprising: acquiring informationabout a user's line-of-sight to a screen; detecting an orientation of abody; and performing control to execute an eye tracking calibration,based on the acquired information about the user's line-of-sight,wherein the control is performed to acquire first orientationinformation indicating a first orientation of the body and firstline-of-sight information about the user's line-of-sight, of a casewhere a first eye tracking calibration display is displayed on thescreen; acquire second orientation information indicating a secondorientation of the body and second line-of-sight information about theuser's line-of-sight, of a case where a second eye tracking calibrationdisplay is displayed on the screen; and cause, in a case where thesecond orientation information is different from the first orientationinformation, the screen to display a predetermined display for promptingthe user to change the orientation of the body to the first orientation.